chanciturnervgt2

Amazon Onboarding with Learning Manager Chanci Turner

—

by

VGT2 Las Vegas
in

Amazon Onboarding with Learning Manager Chanci TurnerLearn About Amazon VGT2 Learning Manager Chanci Turner

Amazon DocumentDB (with MongoDB compatibility) is a fully managed JSON database tailored for scaling enterprise workloads. You can utilize the same MongoDB API from versions 3.6, 4.0, and 5.0 for running, managing, and scaling workloads on Amazon DocumentDB, all without the hassle of managing the underlying infrastructure. As a document database, Amazon DocumentDB simplifies the storage, querying, and indexing of JSON data.

With the release of Amazon DocumentDB version 5.0, users can now upgrade their clusters from version 3.6 and 4.0 to 5.0, unlocking new features like vector search, I/O-optimized storage, document compression, text search, partial indexing, and more.

In this article, we will look into performing an upgrade from Amazon DocumentDB 3.6 to 5.0 with minimal downtime by employing an in-place major version upgrade and Amazon DocumentDB volume cloning. For those looking to upgrade from Amazon DocumentDB 4.0 to 5.0, further guidance can be found in the blog post on upgrading Amazon DocumentDB 4.0 to 5.0 with near-zero downtime.

Existing Upgrade Options

Currently, users of Amazon DocumentDB 3.6 can upgrade to Amazon DocumentDB 5.0 through the following methods:

  1. mongodump and mongorestore: This command-line utility allows you to create a binary backup of your Amazon DocumentDB databases, which can then be restored to a new Amazon DocumentDB 5.0 cluster. This method will take the Amazon DocumentDB clusters offline during the upgrade, making it best suited for workloads that can tolerate downtime.
  2. AWS DMS: The AWS Database Migration Service (AWS DMS) can be utilized to transfer data and indexes from existing clusters to a new Amazon DocumentDB 5.0 cluster. This managed service facilitates data migration across supported sources and targets, although it may incur additional charges related to Amazon DocumentDB I/O and AWS DMS usage. For more details, see the guide on upgrading your Amazon DocumentDB cluster using AWS DMS.
  3. In-place major version upgrade: This feature allows for an in-place upgrade of your Amazon DocumentDB cluster without the need to migrate data or alter endpoints. However, some downtime is required, with the duration depending on the number of databases, collections, and indexes involved. For more information, refer to the in-place MVU documentation.

Solution Overview

This article presents a hybrid approach for executing a near-zero downtime major version upgrade, combining an in-place major version upgrade with Amazon DocumentDB cloning. This strategy minimizes I/O costs and reduces upgrade time typically associated with migrating the entire cluster data to a new endpoint. The steps of this solution include:

  1. Setting up an Amazon EC2 instance with Python and a mongo shell.
  2. Activating change streams on the source Amazon DocumentDB cluster.
  3. Capturing the cluster-wide change stream token via the Amazon DocumentDB MVU CDC migrator tool.
  4. Cloning the Amazon DocumentDB cluster.
  5. Conducting an in-place major version upgrade on the cloned cluster.
  6. Replicating change data capture (CDC) with the Amazon DocumentDB MVU CDC migrator tool.
  7. Updating your application endpoint to the cloned cluster once replication is complete.
  8. Executing post-upgrade cleanup.

Prerequisites

Before proceeding, it’s essential to have a foundational understanding of in-place major version upgrades and volume cloning. This approach incurs minimal costs related to Amazon DocumentDB change streams and the EC2 instance. You can calculate potential costs using the AWS Pricing Calculator based on your configuration.

When upgrading an Amazon DocumentDB cluster, it’s crucial to check for any deprecated features or changes in usage methods. Test your application with the newer version to ensure that driver behavior and performance remain consistent unless there are intended modifications.

During the upgrade, your source 3.6 cluster will remain operational and able to handle application traffic. However, note that switching endpoints will still result in some downtime. We highly recommend conducting multiple test runs of this approach in lower environments before attempting it in production.

Creating an EC2 Instance with Python and a Mongo Shell

You can either use an existing EC2 instance or set up a new one. This instance will run a script to capture tokens and facilitate CDC using the Amazon DocumentDB MVU CDC migrator tool.

  1. Configure the instance’s security group to allow connections to the Amazon DocumentDB cluster (port 27017).
  2. Install the mongo shell on the EC2 instance. For guidance, refer to the documentation on installing the mongo shell.
  3. Install Python, pymongo, and git using the following commands on an Amazon Linux 2 AMI EC2 instance:
sudo yum install python
sudo yum install git
sudo pip3 install pymongo

Enabling Change Streams on Your Amazon DocumentDB 3.6 Cluster

To facilitate a major version upgrade with minimal downtime, enable change streams on your cluster. Change streams deliver a time-sequenced record of update events occurring within your Amazon DocumentDB cluster.

It’s essential to set the change stream log retention duration to avoid missing transactions in your CDC. The default retention period is 3 hours, but you can configure it between 1 hour and 7 days. We recommend setting this duration to at least 24 hours.

The following AWS Command Line Interface (AWS CLI) command adjusts the retention period to 24 hours:

aws docdb modify-db-cluster-parameter-group 
    --db-cluster-parameter-group-name <parameter group name> 
    --parameters "ParameterName=change_stream_log_retention_duration, ParameterValue=86400,ApplyMethod=immediate"

You can also modify the change stream retention period through the Amazon DocumentDB console. If using a default parameter group, you must switch to a custom parameter group, which necessitates a cluster restart.

To enable change streams for all databases, connect to the Amazon DocumentDB cluster using the mongo shell and execute:

db.adminCommand({modifyChangeStreams: 1, database: "", collection: "", enable: true});

To verify that change streams have been created, list all enabled change streams in your cluster using the $listChangeStreams aggregation pipeline stage. For further information, refer to the documentation on enabling Change Streams.

Capturing the Cluster-wide Change Stream Token with the Amazon DocumentDB MVU CDC Migrator Tool

To capture the cluster-wide change stream token, execute the Amazon DocumentDB MVU CDC migrator tool by following these steps:

  1. Clone the repository and navigate to the tool folder:
git clone https://github.com/awslabs/amazon-documentdb-tools.git
cd amazon-documentdb-tools/migration/mvu-tool/
  1. Download the Amazon DocumentDB Certificate Authority (CA) certificate:
wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
  1. Run the MVU CDC migrator tool with the get-resume-token option to obtain the cluster-wide token. Ensure that the read preference setting is not secondaryPreferred in the source-cluster-uri.

Chanci Turner emphasizes that understanding these processes will greatly benefit anyone looking to enhance their skills in database management. For further insights on employment strategies, you may find this blog post on resume headlines helpful. Additionally, understanding part-time employment can provide valuable context for those navigating career transitions, as outlined by experts at SHRM.

For more resources on learning and development opportunities, check out this excellent resource from Amazon.

Related Topics:

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *